Depth sensor combining line triangulation and time of flight

ABSTRACT

Optical apparatus includes a projector, which directs a sequence of pulses of radiation toward a scene so as to form one or more lines of the radiation on the scene. A receiver includes an array of single-photon detectors, which output, in response to the radiation that is incident thereon, signals indicative of a time-of-flight of the pulses from the projector to the receiver via the points in the scene, and collection optics, which form an image the scene on the array, including the one or more lines of the radiation, such that each single-photon detector receives the radiation reflected from a corresponding point in the scene. A processor receives the signals output by the sensing elements, and derives depth coordinates of the points in the scene from both the time-of-flight of the pulses and triangulation of the one or more lines in the image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/700,231, filed Sep. 11, 2017, which claims the benefit of U.S.Provisional Patent Application 62/396,838, filed Sep. 20, 2016, which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to optical systems, andparticularly to optical depth mapping.

BACKGROUND

Existing and emerging consumer applications have created an increasingneed for real-time three-dimensional (3D) imagers. These imagingdevices, also commonly known as depth sensors, 3D mappers, or depthmappers, enable the remote measurement of distance (and often intensity)of each point on a target scene—so-called target scene depth—byilluminating the target scene with one or more optical beams andanalyzing the reflected optical signal.

A commonly used technique for determining the distance to each point onthe target scene involves sending a pulsed optical beam towards thetarget scene, followed by the measurement of the round-trip time, i.e.time-of-flight, taken by the optical beam as it travels from the sourceto target scene and back to a detector adjacent to the source.

Another commonly used technique is based on projecting a pattern ofstructured light onto a scene and capturing an image of the projectedpattern. The pattern may comprise, for example, single or multiple spotsof light or single or multiple lines of light. The pattern may be eitherstationary or scanned across the target scene. The distance to eachpoint in the scene is derived from the local displacement of thepattern.

SUMMARY

Embodiments of the present invention that are described herein provideimproved apparatus and methods for optical depth sensing.

There is therefore provided, in accordance with an embodiment of theinvention, optical apparatus, including a scanning line projector, whichis configured to scan a line of radiation across a scene. A receiverincludes an array of sensing elements, which are configured to outputsignals in response to the radiation that is incident thereon, andcollection optics configured to image the scene onto the array, suchthat each sensing element receives the radiation reflected from acorresponding point in the scene. A processor is coupled to receive thesignals output by the sensing elements, to identify respective times ofpassage of the scanned line across the points in the scene by comparinga time-dependent waveform of the signals from the corresponding sensingelements to an expected waveform, and to derive depth coordinates of thepoints in the scene from the respective times of passage.

In one embodiment, the processor includes distributed processingcomponents that are integrated with the sensing elements.

In some embodiments, comparing the time-dependent waveform of thesignals to the expected waveform includes comparing a temporal durationof the signals to an expected temporal duration. Additionally oralternatively, comparing the time-dependent waveform of the signals tothe expected waveform includes calculating a correlation value betweenthe time-dependent waveform of the signals and the expected waveform.

In a disclosed embodiment, the processor is configured to derive thedepth coordinates of the points in the scene by triangulation, using therespective times of passage and angles of transmission and reception ofthe radiation from the scanning line projector to the sensing elementsvia the scene.

In some embodiments, the sensing elements include single-photondetectors. Typically, the scanning line projector is configured to emitthe radiation as a sequence of short pulses having a pulse durationshorter than the expected waveform, and the signals output by thesensing elements are indicative of a time-of-flight of the pulses fromthe scanning line projector to the receiver via the points in the scene.In one embodiment, the processor is configured to derive the depthcoordinates of the points in the scene from both the respective times ofpassage of the scanned line across the points in the scene and thetime-of-flight.

In a disclosed embodiment, the scanning line projector is configured toscan multiple parallel lines of radiation across respective parts of thescene, and the processor is configured to receive and process thesignals output by the sensing elements in order to identify therespective times of passage of the multiple lines across the points inthe scene.

Typically, the scanning line projector is configured to scan the line ofradiation across the scene in a scan direction perpendicular to theline.

In some embodiments, the processor is configured to construct a depthmap of the scene using the derived depth coordinates.

There is also provided, in accordance with an embodiment of theinvention, a method for sensing, which includes scanning a line ofradiation across a scene, and imaging the scene onto an array of sensingelements, such that each sensing element receives the radiationreflected from a corresponding point in the scene. Signals output by thesensing elements in response to the radiation that is incident thereonare received, and respective times of passage of the scanned line acrossthe points in the scene are identified by comparing a time-dependentwaveform of the signals from the corresponding sensing elements to anexpected waveform. Depth coordinates of the points in the scene arederived from the respective times of passage.

The present invention will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic side view of an optical apparatus, in accordancewith an embodiment of the invention;

FIG. 2 is a schematic representation of the relationship between targetscene depth, time stamps, and direction of scan of a line of radiation,in accordance with an embodiment of the invention;

FIG. 3 is a schematic representation of a signal sensed by a sensingelement, in accordance with an embodiment of the invention; and

FIG. 4 is a schematic representation of a signal sensed by a sensingelement, in accordance with another embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

Depth sensors measure the distance to each point on a target scene(target scene depth) by illuminating the target scene with one or moreoptical beams from a primary light source and analyzing the reflectedoptical signals. The terms “light” and “optical,” as used in the contextof the present description and in the claims, refer to optical radiationin any of the visible, infrared, and ultraviolet ranges. The opticalbeams may comprise either sheets of light, for example, forming one ormore line patterns on the target scene, or beams forming individualspots or spot patterns on the target scene.

Using static line patterns requires a sequence of line patternsprojected one after the other, with an image captured for each linepattern (for example, 10 line patterns and images). The drawbacks ofthis method are the spread of the illumination over a large area,leading to a low signal-to-noise ratio (SNR), and the requirement for alarge number of images, leading to low speed of the 3D mapping. The useof spot patterns suffers, like static line patterns, from spreading ofthe illumination over a large area. An additional drawback is that ofedge artifacts, due to a subset of the spot pattern landing on an edgediscontinuity in the scene.

Using a scanning line concentrates the illumination on a small area,thus improving the SNR. However, this method suffers from sensitivity toboth background noise and to spurious signals from the target scene.

The embodiments of the present invention that are described hereinaddress the above limitations so as to enable fast and robust 3D mappingof target scenes. In the disclosed embodiments, a target scene isilluminated by a scanning line projector. The projector generates a lineof radiation on the target scene, which is scanned continuously acrossthe scene. Alternatively, the projector may be configured to generateand scan multiple parallel lines simultaneously. In the embodimentsdescribed below, the line of radiation is scanned across the scene in adirection perpendicular to the line, but the techniques described hereinmay alternatively be adapted for non-perpendicular scan directions, aswell. The target scene is imaged by a receiver, comprising collectionoptics and an array of sensing elements.

The time-dependent waveforms of the signals from the sensing elementsare monitored by a processor and compared to a waveform that would beexpected from a line passing a sensing element. (The processor thatperforms this function may be a separate unit, or it may be integratedwith the sensing elements, or it may include components that areintegrated with the sensing elements and another component separate fromthe sensing elements.) The processor thus identifies respective times ofpassage of the scanned line across the points in the scene and derivesdepth coordinates of the points in the scene from the respective timesof passage.

Specifically, when the comparison performed by the processor indicatesthat a line image has passed across a given sensing element, a timestamp is generated by the processor, indicating the exact time when thepassage occurred. Knowing the time-dependent angular direction of thebeam generating the line on the target scene, the time stamp enables theprocessor to determine the direction of the beam at the moment marked bythe time stamp. The angular direction of the beam that generated theline image passing the given sensing element is determined by theprocessor from the optical geometry of the receiver. Based on these twoangular directions and the known baseline between the projector and thereceiver, the processor utilizes triangulation to determine the depthcoordinates of the point of the target corresponding to the givensensing element. The comparison of the time-dependent waveform of thesignals from the sensing elements to an expected line waveform ensures arobust identification of signals from the scanned line, and in this wayenables an effective discrimination of the scanned line againstbackground noise or spurious signals from the target scene.

In another embodiment, the projector scans two or more parallel lines atdifferent angles in order to shorten the overall scan time.

Additionally or alternatively, the processor may control the scanner soas to synchronize the scan with a rolling shutter of the array ofsensing elements.

In one embodiment, the processor compares waveforms by measuring thetemporal duration of the time-dependent waveform of the signal receivedby a sensing element. The temporal duration may be determined, forexample, from the time-difference between the leading and trailing edgesof the pulse-shaped signal output by the sensing element. This temporalduration is compared to that expected from a line passing the sensingelement. If the measured temporal duration matches the expected durationto within a predetermined margin, for example ±10%, the signal is deemedto originate from a passing line, and a time stamp is generated for usein triangulation. As background noise or spurious signals from thetarget scene do not in general generate signals within this range ofduration, they are rejected.

In another embodiment, a correlation value is calculated between thetime-dependent waveform of the signal and the expected line waveform. Ifthe correlation value exceeds a predetermined threshold, for example90%, the signal is again deemed to originate from a passing line, and atime stamp is generated for use in triangulation. Similarly to theprevious embodiment, background noise or spurious signals from thetarget scene will in general have a very low correlation value with theexpected line waveform, and they are rejected.

The above two embodiments are described here by way of example, andalternative criteria for waveform comparison that utilize at least twopoints on the time-dependent waveform of the signals from the sensingelements are considered to be within the scope of the present invention.

In some embodiments, the sensing elements in the array comprise fastphotodiodes, with a bandwidth substantially greater than the rate ofscan of the line of radiation across the sensing elements.

In one of these embodiments, the sensing elements in the array comprisesingle-photon detectors, such as single-photon avalanche diodes (SPADs),and the scanning line projector emits the light as high-frequencypulses. The time-dependent waveform of the signal sensed by a givensensing element in response to a passing line of radiation now comprisestwo components:

-   -   1. An envelope due to the shape of the line, and    -   2. Short pulses from the pulsed projector emission, wherein the        sensed pulses are shifted from the emitted pulses by the time of        flight of the optical radiation from the projector via the        target scene to the receiver. As will be detailed below, several        short pulses are typically contained within the envelope.

This embodiment enables the processor to construct a depth map withenhanced accuracy and reliability. The envelope of the signal iscompared to an expected line waveform and a time stamp is generated, asdescribed above, to be used in determining depth coordinates bytriangulation. The shift of the short pulses, when compared to thepulses emitted by the scanning light projector, is utilized by theprocessor in calculating the target scene depth by the method oftime-of-flight. The accuracy of triangulation is proportional to D²,wherein D is the distance to the target point in the scene, whereas theaccuracy of the method of time-of-flight is proportional to D. For shortdistances, triangulation is more accurate, but there is a cross-overpoint, beyond which the method of time-of-flight is more accurate. Inone embodiment, the processor is configured to take advantage of thecombination of the two methods, and to switch between them based on thedistance to points in the scene.

As noted earlier, in some embodiments, the comparison of signals,calculation of correlations, and generation of a time stamp, as well ascalculations of time-of-flight when appropriate, are performed in acentralized processor. In other embodiments, each sensing element in thearray is configured to implement some or all of these functionalities.Such distributed processing assures that the array generates onlysignals based on passage of a line (so-called event-based signals), andthus reduces the volume of transferred data and number of calculationsas compared to embodiments utilizing only a central processor.

System Description

FIG. 1 is a schematic side view of an optical apparatus 20 with ascanned line of optical radiation, in accordance with an embodiment ofthe invention. Optical apparatus 20 comprises a scanning line projector22, a receiver 24, and a processor 25. Scanning line projector 22comprises a light source 26, which comprises a laser diode and opticsfor shaping the laser beam into a sheet of light 33 (components notshown), and an angular scanner 28, such as a rotating mirror. Receiver24 comprises collection optics 30 and an array of sensing elements 32.Processor 25 is coupled to light source 26, angular scanner 28, andarray of sensing elements 32. Although processor 25 is shown, for thesake of simplicity, as a separate, unitary element, in practiceprocessor 25 may comprise distributed processing components that areintegrated with the sensing elements in array of sensing elements 32.

Projector 22 projects sheet of light 33 via angular scanner 28 togenerate a sheet of light 34, which is scanned by angular scanner 28 ina direction perpendicular to the plane defined by sheet of light 34.Sheet of light 34 impinges on a target scene 36, forming on the targetscene a line of light 37 (extending in a direction perpendicular to thepage and thus seen here in cross-section), which is swept, in adirection perpendicular to the line, across target scene 36. A portionof the projected light is reflected towards receiver 24. Target scene 36is shown here, for the sake of simplicity, as an abstract flat surface,but in general, the target that is mapped has a more complex andpossibly dynamic topology. The light reflected from target scene 36 isrepresented by a beam 38; in reality the reflection may be eitherspecular or diffuse. The operation of scanner 28 causes an image of line37 to sweep across array of sensing elements 32. Thus, at theillustrated moment, the image of line 37 is formed on a sensing element40 in array of sensing elements 32.

Processor 25 monitors continuously the time-dependent waveform of thesignals output from all the sensing elements of array of sensingelements 32 and compares them to the waveform that is expected to arisefrom sweeping the image of line 37 across the array. For robust andreliable detection, multiple points in the output signals are used inthe comparison together with multiple points in the expected waveform.Typically, both the rising and trailing sides of the waveform are usedin this comparison.

When processor 25 detects, for example, that a line has swept pastsensing element 40, it generates a time stamp indicating the time ofpassage. The time stamp is used, along with knowledge of the temporalbehavior of the sweep, to determine the directional angle β of lightsheet 34 that generated the line image at sensing element 40. Theposition of sensing element 40 within array of sensing elements 32,together with the optical layout of receiver 24, determines thedirectional angle α of beam 38. Utilizing the values of the twodirectional angles α (for reception of beam 38) and β (for transmissionof sheet 34), together with the known length of a baseline 42 separatingscanning line projector 22 and receiver 24, processor 25 calculates bytriangulation the distance to the point in target scene 36 that gaverise to the signal output by sensing element 40.

In another embodiment (not shown in the figures), multiple sheets oflight are emitted by scanning line projector 22, and multiple parallellines are thus generated and scanned together simultaneously over scene36, in a direction perpendicular to the lines. The multiple sheets oflight can be generated, for example, by multiple light sources or by asingle light source with suitable beamsplitting optics. Typically, theangles between the sheets of light are selected so that scanner 28 scanseach beam over a different part of the scene. Processor 25 receives andprocesses the signals due to all of the projected lines concurrently,thus shortening the overall scan time.

FIG. 2 is a schematic representation of the relationship between targetscene 36 depth, time stamps, and direction of light sheet 34, inaccordance with an embodiment of the invention. Three differentpositions in target scene 36 are shown as an example: positions 36 a, 36b, and 36 c. By observing the time-dependent waveform of the signal atsensing element 40, three different time stamps t_(a), t_(b), and t_(c),are generated from the time-dependent waveforms shown on the right,corresponding, respectively, to target scene positions 36 a, 36 b, and36 c. Each time stamp can now be associated with, respectively, one oflight sheets 34 a, 34 b, and 34 c, and the distance to each depth oftarget scene can be calculated by triangulation.

FIG. 3 is a schematic representation of a signal sensed by a sensingelement in array 32, in accordance with an embodiment of the invention.The time-dependent waveform of the signal comprises a pulse 44,representing the response of the sensing element to a line of lightpassing across the sensing element. The waveform in this example alsoincludes a slowly varying background signal 46 and a discontinuity 48due to, for example, a change in the reflection from the scene.

The comparison of the signal shown in FIG. 3 to the expected waveformfrom a line due to passage of the image of line 37 (FIG. 1) across thesensing element will identify pulse 44 as a signal from a passing line,and will generate time stamp t₁. For example, processor 25 may detectboth the rising edge and the falling edge of pulse 44 and may thusderive the duration of the pulse from the difference in time between therising and falling edges (for example, between the full width at halfmaximum points), for comparison with the expected pulse duration.Alternatively or additionally, processor 25 may calculate across-correlation between pulse 44 and the expected waveform. When thewidth and/or the correlation satisfy a predefined threshold criterion,processor 25 generates the corresponding time stamp at the time ofoccurrence of pulse 44. However, discontinuity 48, when compared to anexpected line waveform, will not pass the comparison test, will not beidentified as a line passage, and will consequently be ignored.Background signal 46 is similarly ignored.

FIG. 4 is a schematic representation of a time-dependent waveform of asignal 50 sensed by a sensing element, in accordance with anotherembodiment of the invention. In this embodiment (referring to theelements shown in FIG. 1), light source 26 comprises a laser, whichoutputs a sequence of very short pulses of light (typicallysub-nanosecond), so that sheet of light 34 comprises pulsed radiation,and array 32 comprises SPAD sensing elements. Signal 50 in FIG. 4represents a time-dependent waveform generated by a SPAD sensing elementin array 32 under these pulsed illumination condition. The amplitudeenvelope of signal 50 (not drawn explicitly) is due to the width of line37 that is imaged onto the SPAD sensing element, while pulses 52 underthe envelope are due to the pulsed light sensed by the SPAD.

The time stamp t₁ in this example is derived from the width of theenvelope of signal 50. This time stamp is utilized in a triangulationcalculation by processor 25 to find the target depth, as previouslydescribed. Additionally or alternatively, processor 25 performstime-of-flight-based depth calculation by comparing the temporalposition of individual pulses 52 to the timing of the pulses emitted bylight source 26. The temporal offset between the emitted pulses andreceived pulses 52 gives the time-delay due to the travel of the pulsesfrom scanning line projector 22 via target scene 36 to receiver 24, andthus the depth of the scene point from which the light pulses reflected.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and subcombinationsof the various features described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art.

The invention claimed is:
 1. Optical apparatus, comprising: a projector,which is configured to direct a sequence of pulses of radiation toward ascene so as to form and scan multiple lines of the radiationsimultaneously across respective parts of the scene; a receivercomprising an array of single-photon detectors, which are configured tooutput, in response to the radiation that is incident thereon, signalsindicative of a time-of-flight of the pulses from the projector to thereceiver via the points in the scene, and collection optics configuredto form an image the scene on the array, including the multiple lines ofthe radiation, such that each single-photon detector receives theradiation reflected from a corresponding point in the scene; and aprocessor coupled to receive the signals output by the single-photondetectors, and to derive depth coordinates of the points in the scenefrom both the time-of-flight of the pulses and triangulation of themultiple lines in the image based on an envelope of a time-dependentwaveform generated by the signals output by the single-photon detectorsin response to the sequence of the pulses as the lines scan across thescene.
 2. The optical apparatus according to claim 1, wherein theprocessor comprises distributed processing components that areintegrated with the single-photon detectors.
 3. The optical apparatusaccording to claim 1, wherein the single-photon detectors comprisesingle-photon avalanche diodes (SPADs).
 4. The optical apparatusaccording to claim 1, wherein the processor is configured to construct adepth map of the scene using the derived depth coordinates.
 5. Theapparatus according to claim 1, wherein the projector is configured toscan the multiple parallel lines in a direction perpendicular to thelines.
 6. A method for sensing, comprising: directing a sequence ofpulses of radiation toward a scene so as to form and scan multiple linesof the radiation simultaneously across respective parts of the scene;forming an image of the scene on an array of single-photon detectors,which are configured to output, in response to the radiation that isincident thereon, signals indicative of a time-of-flight of the pulsesvia the points in the scene, such that each single-photon detectorreceives the radiation reflected from a corresponding point in thescene; receiving signals output by the single-photon detectors inresponse to the radiation that is incident thereon; and processing thereceived signals so as to derive depth coordinates of the points in thescene from both the time-of-flight of the pulses and triangulation ofthe multiple lines in the image based on an envelope of a time-dependentwaveform generated by the signals output by the single-photon detectorsin response to the sequence of the pulses as the lines scan across thescene.
 7. The method according to claim 6, wherein processing thereceived signals comprises processing the signals using distributedprocessing components that are integrated with the single-photondetectors.
 8. The method according to claim 6, wherein the single-photondetectors comprise single-photon avalanche diodes (SPADs).
 9. The methodaccording to claim 6, wherein processing the received signals comprisesconstructing a depth map of the scene using the derived depthcoordinates.
 10. The method according to claim 6, wherein the multipleparallel lines are scanned in a direction perpendicular to the lines.